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Abstract 

This note proposes a method, which can be applied to searches and more in 
general to any cross section measurement, to maximize the analysis sensitivity. 


1 Introduction 


During the last years at LEP a new attitude toward searches has developed which tends to 
separate the actual search for new phenomena from the derivation of an exclusion limit. In this 
note we would propose a new method of globally optimized analysis which combines the two 
steps in an optimal and unbiased way by making use of background subtraction. 

2 Statistical Issues 

Several methods have been proposed at LEP for the extraction of limits on the Higgs boson 
mass H and relative selection optimization Q, while for most of other searches the limits 
are based on the PDG formula [|J. No methods, to our knowledge, have been proposed yet 
to optimize an analysis aimed at discovery and deriving the related discovery confidence level 
(CL). 

When measuring physical quantities the combination of different results with gaussian errors 
is straightforward. On the other hand it is usually more complicated to combine different 
exclusions or discovery CL. 

Clearly to derive a discovery CL the background has to be subtracted. Therefore, aiming at 
a globally optimized analysis capable of discoveries and exclusions at the same time we should 
definitely consider background subtraction. 

To make a CL evaluation there are two possibilities: 

• either one has a formula capable of calculating the CL of the outcome of a given experi¬ 
ment 

• or one defines an estimator to rank the experiments from the more signal (background)- 
like to the less signal(background)-likc and then, either analytically or by use of Monte 
Carlo (MC) techniques, evaluates the ranking of the actual experiment. 

Some formulae have been proposed for the Erst case as some CL evaluation methods have 
been proposed for the second case. We would like to note that in the second case the estimator 
can be built arbitrarily (provided that it is done a-priori). 

In all these approaches the only thing which is used to derive the CL is the signal to 
background ratio in the points where the candidates are observed. 

The formula in Jl[] , derived in the Bayesian approach, is very simple and it naturally includes 
background subtraction and therefore in the following we will use it. As it has not yet been 
demonstrated that this formula gives a statistically well defined CL we use its output as an 
estimator which will then be treated with MC methods. It should be noted at this point that the 
only difference between the methods is due to the different estimator chosen. Our choice, which 
is anyway not essential for the results of the method, is due to the simplicity and naturalness 
of the estimator described in [|l|] and also to the fact that its output may sometimes be a little 
conservative but it is a very good approximation of the CL. 

The MC method has on the other hand the big advantage that it allows, with virtually 
no additional effort, to include any sort of statistical and systematic errors on efficiency and 
background when computing the CL. 
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3 Confidence Levels 


For sake of clarity we try in this section to explain in simple words the definition of confidence 
level and the different methods used to derive it. 

The results of a measurement are compared with a given hypothesis, for instance that the 
Higgs boson has a given production cross section, and the confidence level associated to the ex¬ 
clusion of our hypothesis gives the probability that the disagreement between the measurement 
and the given hypothesis is smaller than what actually observed. 

As previously stated we can either do an analytical calculation to derive the confidence level 
for a given experiment, or we can evaluate it with Monte Carlo technique. Unfortunately the 
first possibility seems not to be viable because it is virtually impossible to obtain an analytical 
expression to relate any given estimator, that we want to use to rank the experiment in the 
most effective way, with its probability density function. 

The cross check of the analytical calculation of the confidence level can be done by means 
of Monte Carlo simulations of trial experiments. In practice this means that we perform our 
measurements many times, introducing each time a number of background and signal events 
according to a Poisson distribution with mean equal to the sum of the expected background 
and signal (our hypothesis) levels. 

The actual probability associated to the calculated confidence level is then simply given by 
the rate of simulated measurements in larger disagreement with the given hypothesis than the 
real one, properly taking into account the fact that the background alone must give a value of 
the estimator smaller than the observed one. The latter means that the background can not 
fluctuate over the actual number of events observed in the experiment and that its probability 
distribution is bound from above. If we do not take into account that, as already reported 
in Reference 0, we can derive the absurd conclusion that every signal, even with null cross 
section, can be excluded with extreme over-fluctuation of the background. 

It is clear that we can use this Monte Carlo cross check directly as a definition of the 
confidence level. This turns out to be a powerful tool because we can use what we believe the 
most sensitive estimator to rank the measurement and then the probability associated to it is 
simply obtained as previously explained. 


4 The Estimator 


We would first of all like to stress that the estimator we used [IJ is a very simple extension of the 
PDG prescription to compute CL for experiments with or without background subtraction || 
to the multichannel situation: 


F( Sl b,n)=e-^l[^' fj + b ' 9j)n3 

3 = 1 


Ujl 


where the index j runs over the different channels and where f 3 and g 3 are the fraction of all 
signal events (s) and all background events (6), which are in the channel j. The fraction of the 
total number of observed candidates n, in each channel j , is rij. Using the Bayesian approach 
we can obtain the confidence level for a given number of expected signal events a with the 
following integrations: 


V c 


J?F(s,b,n)ds _ 
fo° F(s, b, n)ds — 
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Vc is the estimator we use to rank the experiment. 

Using it as an estimator, it is straightforward to extend it to be without background sub¬ 
traction. It is enough to compare the estimator with the distribution of signal-only random 
experiments. 

To get the correct CL we compare the actual value of the estimator to the distribution of 
trial MC experiments using signal and background. Clearly, given the fact that the experiment 
excludes the possibility of having a background which gives by itself a larger estimator than the 
measured one we only consider those MC experiments with the background estimator smaller 
than the measured estimator. We add to those a random signal, then measure the value of 
the probability. Usually, as shown below, the true CL is slightly different from the one we can 
obtain from equation [l]. The CL calculated with equation []]. for a small number of events, is 
always conservative. For a large number of events, it tends to be a correct estimation of the 
actual CL measured with MC experiments. 


5 Analysis Optimization 

As said in Section |2] all methods for limits calculation proposed for a single channel are only 
sensitive to the signal/background ratio in the point or analysis (or whatever) where the data 

lie. 

The new method proposed here is to consider this fact and its direct consequence that an in¬ 
trinsically optimized analysis can be obtained when for any value of the ratio signal/background 
we obtain the highest possible overall efficiency. 

It is important to understand that the specific power of a subanalysis only depends on 
the signal to background ratio and not on the absolute amount of the signal selected by the 
subanalysis. Then the optimization can be achieved including all the available information 
provided that channels/analyses with different signal/background ratio are not mixed. This 
means that we have to organize a set of complementary analyses (i.e. each one selects events 
that no other analysis selects) and then whatever analysis we add the overall sensitivity is 
improved. 

If we are able to devise such an optimal selection, which orders the different subselections in 
terms of signal to background ratio, whatever estimator we use, the analysis can be optimized 
by a single cut on the signal to background ratio. The latter in practice is often needed to take 
into account systematic uncertainties. 

To illustrate these concepts, let us consider a simple numerical example. Let us take the 
usual case of the search for the production of a new particle which can occur in two channels 
A and B with branching ratios of 20% and 80% respectively. Let us assume that in the two 
channels we have the same selection efficiency of 50% with a mass resolution for the signal of 
0.3 in channel A and 0.1 in channel B, and that we have a flat background of 1 event from 
-1 to 1 (we take as an example the mass but it can be any other variable). Binning such a 
distribution in bins of width of 0.1, the result is shown in figure |T|-a for channel A and [L]-c for 
channel B. 

These two spectra can be rearranged as in figure [T]-b for channel A and [T[-d for channel B, 
where the bins of the two analyses are ordered in signal/background ratio. The combination of 
the first two analyses, shown in figures Q-b and GK is represented by the dots in figure ^j. 

At this point if we increase the signal efficiency we can improve the overall sensitivity. For 
instance we take two additional selections, C and D, both with 15% additional efficiency on 
channel A and B respectively. For simplicity the mass resolution and the background level for 
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Figure 1: Distributions for signal and background in analyses A (a) B (c) C (e) D (g). Equiv¬ 
alent distributions, analyses A (b) B (d) C (f) D (h), with signal/background ratio ordering. 
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Figure 2: Combination of analyses A B C D (histogram) and of analyses A and B only (dots). 
Analysis bins are ordered according to their signal/background ratio. 
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the new analyses are assumed to be the same as those in the original analyses A and B. The 
equivalent plots are also shown in figures HJ-e-f for analysis C and in figures |l]-g-h for analysis 
D. 

From the combination of the four analyses, shown in figure |j. it is apparent that the A-B- 
C-D analyses combined are better than A-B analyses alone. As it will be explained in more 
detail in the following it is clear that, depending on the estimator used and on the systematic 
errors, it may be useful to reject the bins with worse signal/background ratio. In conclusion, to 
obtain a globally optimized analysis for exclusion and discovery at the same time it is sufficient 
to have an optimized ordering of the analyses, or of the analysis bins, which can be obtained 
with the best available variables. 

LEP experiments so far for the calculation of Higgs limits have used estimators which 
evaluate the signal/background ratio by means of the reconstructed Higgs mass. This is not 
always the best information to discriminate signal from background (for instance in the H°qq 
channel the b-tagging is much better) and in any case is not the only one. 

One possibility, for example, is to use neural network and the neural network output. Another 
is to use simple selections cuts linked together by means of a global variable as previously shown 
in U . The details of this last technique will be described in a dedicated paper 0 . 

According to formula |5|], where background subtraction is used, adding a new channel, even 
with very poor signal/background ratio, the average analysis sensitivity does not worsen. 

This statement implies that all channels should be used because they improve the overall 
sensitivity, even if very little. Unluckily this is an ideal situation because, as we will see in the 
following, uncertainties on the background level, both from statistical and systematic sources, 
do not allow the use of analyses with very poor signal/background ratio. So the optimal 
analysis should have a cut on the signal/background ratio according to the amplitude of the 
uncertainties. 

In particular as we have already seen it is better not to join in a single analysis channels 
with different signal/background ratio. Furthermore we split every single channel/analysis into 
several complementary analyses with different signal/background ratios, and we try to do that 
in the most effective way. But we would like to warn anyway that one should be cautious 
with that procedure because it may lead, if we do not take into account uncertainties on the 
background level, to average and sometimes also actual limit overestimation due to large MC 
statistical errors in the different bins of the analysis. On the other hand if we take into account 
properly the uncertainties on the background level and these are too high, we are then obliged 
to apply a tight cut on the signal/background ratio. 

Concluding the definition of the optimal analysis also depends on the amount of MC systematic 
uncertainties. 

To better explain this second important issue of our method, i.e. how many analyses and how 
much background should be accepted, we will treat in the following section a simple example, 
rather than describing the application of this method to Higgs searches at LEP 161 -t- 172 as 
reported in [||. 

6 A Simple Example 

Let us now make a simple example, let us consider the case of a binned analysis with differ¬ 
ent efficiencies and background contaminations. The assumed efficiencies and background are 
reported in table |l] and the number of expected events for signal and background are shown 
in figure |3|. Each bin can be considered for example as a different decay channel of the same 
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Figure 3: Number of expected events for signal and background for the different analyses. For 
the signal we assumed 5 events expected. 


Analysis 

Efficiency 

Expected background 

S/B 

1 

0.3 

0.1 

15.000 

2 

0.2 

0.2 

5.000 

3 

0.1 

0.5 

1.000 

4 

0.1 

1.0 

0.500 

5 

0.1 

2.0 

0.250 

6 

0.1 

5.0 

0.100 

7 

0.05 

10.0 

0.025 

8 

0.03 

10.0 

0.015 

9 

0.02 

10.0 

0.010 


Table 1: Efficiency, number of events expected from the background and signal/background 
ratio for nine different analyses. 

















Last analysis included 

Average Vc 

Average Vt 

1 

0.2535 ± .0001 

0.2535 ± .0001 

2 

0.1299 ± .0001 

0.1147 ± .0001 

3 

0.1119 ± .0001 

0.0877 ± .0001 

4 

0.1028 ± .0001 

0.0752 ± .0001 

5 

0.0978 ± .0001 

0.0700 ± .0001 

6 

0.0956 ± .0001 

0.0686 ± .0001 

7 

0.0956 ± .0001 

0.0685 ± .0001 

8 

0.0954 ± .0001 

0.0684 ± .0001 

9 

0.0955 ± .0002 

0.0676 ± .0002 


Tabic 2: Average Vc and Vt for a large number of trial experiments, in the background only 
hypothesis. The first row includes only the first analysis, while following rows include every 
time a new analysis for a total of 9 analyses in the last row. 


physical process (for instance Higgs production). 

6.1 Statistical errors only 

First of all we consider the case of background subtraction with no systematic errors to approach 
the problem only on the statistical side. 

We perform a set of MC experiments with the signal and background indicated for each bin, 
each time adding a new channel. For each experiment we compute Vc as given by equation 
|T] for the two hypotheses: background only and signal plus background. After correction to 
have exact probability we get Vt, whose definition is described below, and we obtain the results 
shown in Figures |4| and ||, where we show the distributions for Vc and for Vt for a large number 
of trial experiments (between 300 thousand and 1 million). 

These Figures are ordered in increasing number of included analyses. The value of Vc is 
directly obtained from equation [[] while the value Vt is obtained with Monte Carlo technique 
in order to have the exact probability. To obtain the exact probability, from MC experiments, 
for each value of our estimator Vc we should count the number of experiments (N exp ) with a 
Vc for background+signal smaller than Vc, where Vc is the value of the estimator obtained 
from the measurement. Clearly, to properly take into account the information contained in the 
experiment under study the background alone should have a Vc smaller than Vc- We should 
then divide by the fraction of experiments satisfying this last relation. Consequently we define 
Vt as: 

p _ Jo V eX p (VCsig+back ) dVcsig-\-back 

Ne X p{VCbackjdPCback 

One indicator of the sensitivity of the experiment is, at this point, the average Vt for 
experiments without signal, reported in table |2] adding the bins of analysis one by one. 

It is apparent that, when dealing only with statistical error and performing background 
subtraction, the sensitivity of the experiment is never degraded by the addition of additional 
analyses regardless of the amount of background that they contribute. It can be seen from 
the average value of Vc and Vt that the gain is remarkable if we include channels with sig¬ 
nal/background ratio as low as 0.1, and is almost negligible for channels with signal/background 
ratio between 0.1 and 0.01. 
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Figure 4: Distributions for Vc in the signal + background (a) and background only (b) hy¬ 
potheses and for Vt in the same hypotheses, (c) and (d), for a large number of trial experiments, 
starting from analysis 1 alone and adding every time another analysis. 
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Figure 5: Distributions for Vc in the signal + background (a) and background only (b) hy¬ 
potheses and for Vt in the same hypotheses, (c) and (d), for a large number of trial experiments, 
starting from the first five analyses and adding every time another analysis. 
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Last analysis included 

Average Vc 

Average Vt 

1 

0.2535 ± .0009 

0.2880 ± .0009 

2 

0.1296 ± .0009 

0.1693 ± .0009 

3 

0.1133 ± .0009 

0.1428 ± .0009 

4 

0.1076 ± .0009 

0.1307 ± .0009 

5 

0.1081 ± .0009 

0.1437 ± .0009 

6 

0.1111 ± .0009 

0.1451 ± .0009 

7 

0.1136 ± .0009 

0.1614 ± .0009 

8 

0.1182 ± .0009 

0.1530 ± .0009 

9 

0.1174 ± .0009 

0.1642 ± .0009 


Tabic 3: Average Vc and Vt for 10000 trial experiments, in the background only hypothesis. 
The first row includes only the first analysis, while following rows include every time a new 
analysis for a total of 9 analyses in the last row. A systematic uncertainty of 40% both on 
signal efficiency and on the background level have been introduced. 


In this study, even if not unambiguously demonstrated, the value of Vc turned out to be 
always higher than the value of the true probability; thus it seems impossible that, using Vc 
as the true confidence level, optimistic limits were obtained. 

6.2 Statistical and systematic errors 

We repeat the same exercise this time taking into account possible systematic errors on the 
signal efficiency and on the background level and we want to check the stability of the method. 

We perform a set of MC experiments with the signal and background indicated for each 
bin, each time adding a new channel. As before for each experiment we compute Vc f° r the 
two hypotheses: background only, signal + background. 

Now in addition we introduce a systematic uncertainty of 40% both on signal efficiency and 
on the background level applied when randomly generating the outcome of the MC experiment. 
This systematic uncertainty is introduced in each MC experiment according to a gaussian 
distribution with mean equal to 40%. 

A systematic uncertainty of 40% is really an extreme case but is useful to check the results 
of the method. From Table |3] we realize that starting from the fifth analysis (corresponding to 
a signal/background ratio of 0.25) the average sensitivity worsens. This is an important result 
since it clarifies that according to the systematic uncertainty we have to reject analyses with a 
signal/background ratio lower than a certain value in order to reach the best sensitivity. 

In Tables [4] and [5| we show the behaviour of the Average Vc and Vt when a systematic 
uncertainty of 20% is considered only for signal efficiency (Table [I]) or for background level 
(Table ||) . 

From Table [| we can not see any striking worsening of the sensitivity while in Table |5| it 
is rather significant the worsening starting from the sixth analysis (signal/background ratio 
equal to 0.1). We can then conclude that systematic errors on the background level are more 
important than those on the signal efficiency. 

In Figure [] we show the average value of Vt as a function of the number of analyses included. 
In this Figure we compare the behaviour of < Vt > for different amplitudes of systematic errors. 
In the same Figure we also show the behaviour of Vt in the case of no background subtraction. 
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Last analysis included 

Average Vc 

Average Vt 

1 

0.2535 ± .0009 

0.2635 ± .0009 

2 

0.1291 ± .0009 

0.1261 ±.0009 

3 

0.1118 ± .0009 

0.1005 ± .0009 

4 

0.1039 ± .0009 

0.0825 ± .0009 

5 

0.0980 ± .0009 

0.0829 ± .0009 

6 

0.0965 ± .0009 

0.0838 ± .0009 

7 

0.0970 ± .0009 

0.0765 ± .0009 

8 

0.0961 ± .0009 

0.0809 ± .0009 

9 

0.0958 ± .0009 

0.0766 ± .0009 


Table 4: Average Vc and TV for 10000 trial experiments, in the background only hypothesis. 
The first row includes only the first analysis, while following rows include every time a new 
analysis for a total of 9 analyses in the last row. A systematic uncertainty of 20% on signal 
efficiency have been introduced. 


Last analysis included 

Average Vc 

Average Vt 

1 

0.2539 ± .0009 

0.2597 ± .0009 

2 

0.1302 ± .0009 

0.1193 ± .0009 

3 

0.1130 ± .0009 

0.0913 ± .0009 

4 

0.1050 ± .0009 

0.0716 ± .0009 

5 

0.0997 ± .0009 

0.0745 ± .0009 

6 

0.1006 ± .0009 

0.0772 ± .0009 

7 

0.1007 ± .0009 

0.0773 ± .0009 

8 

0.1017 ± .0009 

0.0802 ± .0009 

9 

0.1017 ± .0009 

0.0773 ± .0009 


Table 5: Average Vc and Vt for 10000 trial experiments, in the background only hypothesis. 
The first row includes only the first analysis, while following rows include every time a new 
analysis for a total of 9 analyses in the last row. A systematic uncertainty of 20% on the 
background level have been introduced. 


Last analysis included 

Average Vc 

Average “TV” 

1 

0.2532 ± .0009 

.2588 ± .0009 

2 

0.1296 ± .0009 

.1240 ± .0009 

3 

0.1111 ± .0009 

.1087 ± .0009 

4 

0.1028 ± .0009 

.1222 ± .0009 

5 

0.0982 ± .0009 

.1544 ± .0009 

6 

0.0966 ± .0009 

.2125 ± .0009 

7 

0.0971 ± .0009 

.2458 ± .0009 

8 

0.0959 ± .0009 

.2599 ± .0009 

9 

0.0951 ± .0009 

.2811 ± .0009 


Table 6: Average Vc and “TV” for 10000 trial experiments, in the background only hypothesis. 
The first row includes only the first analysis, while following rows include every time a new 
analysis for a total of 9 analyses in the last row. No background subtraction has been applied 
to to compute “TV”- 
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Figure 6: Average value of Vt as a function of the number of analyses included. The five lines 
show the behaviour of < Vt > for different amplitudes of systematic errors and in the case of 
no background subtraction. 
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Last analysis included 

Discovery at < 0.01 

Discovery at < 0.001 

Discovery at < 0.0001 

1 

0.4747 ± 0.2580 

0.2168 ± 0.1381 

0.0786 ± 0.0552 

2 

0.6088 ± 0.0779 

0.3492 ± 0.0414 

0.2183 ± 0.0499 

3 

0.6318 ± 0.0002 

0.3997 ± 0.0071 

0.2366 ± 0.0195 

4 

0.6292 ± 0.0021 

0.3988 ± 0.0027 

0.2312 ± 0.0089 

5 

0.6339 ± 0.0011 

0.3989 ± 0.0076 

0.2261 ± 0.0143 

6 

0.6326 ± 0.0030 

0.3991 ± 0.0092 

0.2063 ± 0.0245 

7 

0.6338 ± 0.0031 

0.3973 ± 0.0090 

0.2453 ± 0.0246 

8 

0.6296 ± 0.0030 

0.3877 ± 0.0144 

0.2237 ± 0.0120 

9 

0.6345 ± 0.0030 

0.4006 ± 0.0101 

0.2254 ± 0.0209 


Tabic 7: Discovery probability at different CL. The first row includes only the first analysis, 
while following rows include every time a new analysis for a total of 9 analyses in the last row. 


This shows that even large systematic errors on the efficiency are not significant while the 
same uncertainties on the background play a role. In the case of huge systematic errors (40%) 
the sensitivity degrades considerably including analyses with high level of background. If one 
has to deal with such a situation it is obviously better to reject analyses with too low signal to 
background ratio (in our example between 0.1 and 0.25). 

Finally in Table ^]we show the values that would be obtained if we neglected the existence 
of background and we compared the outcome of the experiment with signal only MC experi¬ 
ments. If we did that the addition of high background analyses would definitely deteriorate the 
sensitivity as can be clearly seen from Figure ||. 

7 Discoveries 

To estimate the analysis sensitivity for discoveries we proceed exactly in the same way as 
previously shown when optimizing the analysis sensitivity for exclusions. 

The only difference is that we set the discovery CL at the requested probability for the 
measurement to be consistent with the background only hypothesis. 

In Table [7| we show the probability to discover our signal at different levels of significance 
(1%, 0.1% and 0.01%) as a function of the number of analyses included. 

From this Table we can see that the gain due to the inclusion of low purity analyses is 
much less than for the case of exclusion as shown in Table [2]. There we had a remarkable 
gain including analyses with signal/background ratio as low as 0.1, while in Table [7] we have a 
remarkable gain only including analyses with signal/background ratio not lower than 1. 

8 Measurements and other issues 

Using the same estimator described before is possible to directly measure the rate of a given 
process. It can easily be seen that the optimization method here proposed also improves the 
errors on the measurement. 

Another issue which follows naturally from this optimization method is the combination 
of results from different experiments. From what we have described it should be clear that 
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different experiments can be treated on the same footing just as different analyses with their 
own efficiencies, backgrounds and systematic errors. 


9 Conclusions 

We propose a new method, which can be applied to searches and to any cross section measure¬ 
ment. 

Using a large variety of estimators we are able to devise an intrinsically optimized analysis 
which is the same for discovery (and relative cross section measurement) and for exclusion. 

Using this method the sensitivity of the analysis is improved compared to most of previously 
used methods. 
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